Supervised machine learning-based modeling of sensitivities to potential disruptions

ABSTRACT

Computer-implemented systems, methods and products for modeling sensitivities to potential disruptions by observing performances of entities in a first sub-population and a second sub-population using a machine learning model comprising a set of predictors and a binary indicator variable associated with a first entity subjected to a first event associated with the first sub-population, the machine learning model trained to predict an expected performance for the first entity based on at least one of a known attribute associated with the first entity in relation to the first event and a value of the binary indicator variable associated with the first event.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of and claims priority to the earlierfiling date of U.S. patent application Ser. No. 17/469,794, filed onSep. 8, 2021, which is a continuation-in-part of and claims priority tothe earlier filing date of U.S. patent application Ser. No. 15/801,265,filed on Nov. 1, 2017 the content of which is hereby incorporated hereinin its entirety.

TECHNICAL FIELD

The subject matter described herein relates to supervised machinelearning-aided analysis of historical data sets for development ofmodels for quantifying sensitivity of outcome predictions to potentialfuture disruptions.

BACKGROUND

Predictive modeling is broadly applicable in many industries. Inparticular, there is a need for robust modeling that can quantify asensitivity of a given outcome prediction to a potential future binarydisruption. For example, predictions of expected drug efficacy,resilience of a manufacturing process or an energy grid, etc. may notproperly account for unknown sensitivities to occurrence of a futuredisruption. Currently available approaches to quantifying suchsensitivities are either highly speculative or computationally unwieldy.

SUMMARY

This document presents systems, methods, and techniques for developingand using machine learning-based for use in preparing and analyzinghistorical data to generate models for quantifying sensitivity ofpredictions regarding an entity's expected performance to some futurepotential disruption.

In one aspect, a method of training a sensitivity index model forpredicting the sensitivity of an entity to a potential future disruptionincludes identifying a population of entities for which historical dataattributes for each of a plurality of data attributes are availablecontemporaneous to an event that causes a binary disruption in thestatus quo or contemporaneous to an absence of the event or to a lessdisruptive second event, and dividing the population of entities into afirst sub-population and a second sup-population. The firstsub-population experiences the event and the second sub-population doesnot experience the event or experiences the less disruptive secondevent. The dividing includes calculating a propensity score for theentities in the population of entities using supervised machine learningand creating pairs of matched entities in the population of entitiessuch that the propensity score for each entity in the firstsub-population is similar to that of a matched entity in the secondsub-population.

The method further includes modeling observed performances of theentities in the first sub-population and the second sub-population,which includes defining a set of predictors and a binary indicatorvariable for each entity, and producing a model for predicting outcomesbased on an entity's historical data attributes and a value of thebinary indicator variable. Expected performances of the entities in boththe first and second sub-populations are calculated under disrupted andless disrupted conditions using the model, which includes, for eachentity, varying the value of the binary indicator variable while keepingthe entity's historical attribute values fixed.

A sensitivity value is calculated for each entity by quantifying adifference between the calculated expected performance under thedisrupted condition and the expected performance under the lessdisrupted condition, and a sensitivity index model is generated, whichincludes using supervised learning techniques based on the calculatedsensitivity values and the historical attribute values for each entity.

In another aspect, a method for generating a sensitivity index score foran entity of interest includes receiving input attribute values of oneor more of a plurality of historical data attributes for the entity ofinterest and using the input attribute values as model inputs to asensitivity index model trained with a training process as described inthe preceding aspect.

An improved computer-implemented machine learning system is alsoprovided that includes one or more processors coupled to one or moredata storage devices to process or store data associated with knownattributes for a plurality of entities subjected to a plurality of knownevents. A machine learning model implemented as a data structure storedin the one or more data storage devices, the machine learning modeltrained to observe performances of the plurality of entities in a firstsub-population and a second sub-population.

The machine learning model may be associated with a set of predictorsand at least a binary indicator variable corresponding to a first entitysubjected to a first event experience by members of the firstsub-population. The machine learning model may be trained to predict anexpected performance for the first entity based on at least one of (1) aknown attribute associated with the first entity in relation to thefirst event and (2) a value of the binary indicator variable associatedwith the first event.

Members of the first sub-population and the second sup-population aredistinguishable based on whether one or more events have beenexperienced by the members of the first sub-population and the secondsup-population, the members of the first sub-population experiencing thefirst event and the members of the second sub-population experiencingthe second event. An expected performance of the first entity subjectedto the first event is determined based on a sensitivity value determinedfor the first entity by quantifying an expected performance of a memberof the first sub-population subjected to the first event according to asensitivity index generated based on different permutations of potentialoutcomes for the plurality of known events.

A treatment may be applied to the first entity, responsive to thedetermined expected performance. The first-subpopulation and the secondsub-populations optimally capture a binary test of a type of outcome forthe sensitivity index based on different events with different timeperiods associated with different known attributes. The different knownattributes are contemporaneous to at least one of (1) the differentevents causing a binary disruption in the first entity's status quo, (2)an absence of the first event, and (3) the second event being lessdisruptive than the first event. The different known attributes may havevalues that are non-randomized, such that the first entity experiencingthe first event and a second entity experiencing the second eventexhibit materially different distributions of said values.

In one embodiment, the sensitivity index is generated based on at leasta first disruption type or a first target outcome. A known attribute isdetermined based on historical data associated with a population ofentities having been subjected to an event. The population is dividedinto the first sub-population and the second sub-population based on apropensity score calculated for pairs of matched entities in thepopulation. The first event comprises a first medical treatment and thesecond event comprises a second medical treatment not including thefirst medical treatment. The historical data comprises one or more ofage, weight, body mass index, ethnic background, socio-economic factors,and health condition. The sensitivity index predicts how sensitive apredicted health outcome for the first entity is depending on apotential future disruption in the first medical treatment.

Implementations of the current subject matter can include, but are notlimited to, systems and methods including one or more features orperforming one or more operations consistent with the descriptionsherein as well as articles that comprise a tangibly embodiedmachine-readable medium operable to cause one or more machines (e.g.,computers, etc.) to result in such features or operations describedherein. Similarly, computer systems are also described that may includeone or more processors and one or more memories coupled to the one ormore processors. A memory, which can include a computer-readable storagemedium, may include, encode, store, or the like one or more programsthat cause one or more processors to perform one or more operations orprovide one or more features described herein. Computer implementedmethods consistent with one or more implementations of the currentsubject matter can be implemented by one or more data processorsresiding in a single computing system or multiple computing systems.Such multiple computing systems can be connected and can exchange dataand/or commands or other instructions or the like via one or moreconnections, including but not limited to a connection over a network(e.g. the Internet, a wireless wide area network, a local area network,a wide area network, a wired network, or the like), via a directconnection between one or more of the multiple computing systems, etc.

The details of one or more variations of the subject matter describedherein are set forth in the accompanying drawings and the descriptionbelow. Other features and advantages of the subject matter describedherein will be apparent from the description and drawings, and from theclaims. The claims that follow this disclosure are intended to definethe scope of the protected subject matter.

DESCRIPTION OF DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this specification, show certain aspects of the subject matterdisclosed herein and, together with the description, help explain someof the principles associated with the disclosed implementations. In thedrawings,

FIG. 1A is a flowchart illustrating features of a method for developinga sensitivity model, in accordance with aspects described herein;

FIG. 1B is a flowchart illustrating features of a method for developingan ensemble sensitivity model, in accordance with aspects describedherein;

FIG. 2 depicts a block diagram illustrating a computing system, inaccordance with aspects described herein;

FIG. 3 is a diagram of an individual's sensitivity with respect to twodifferent conditions (e.g., a normal and stressed condition), inaccordance with aspects described herein;

FIG. 4 is a time diagram that illustrates a longitudinal study design,in accordance with aspects described herein;

FIG. 5A is a diagram illustrating a difference between an average numberof inquiries for the 20% most economic sensitive and the 20% leasteconomic sensitive consumers within a risk score band, in accordancewith aspects described herein;

FIG. 5B is a diagram illustrating a difference between an average totaltrade line balance for the 20% most economic sensitive and the 20% leasteconomic sensitive consumers within a risk score band, in accordancewith aspects described herein;

FIG. 5C is a diagram illustrating a difference between an average numberof months since the most recent trade line for the 20% most economicsensitive and the 20% least economic sensitive consumers within a riskscore band, in accordance with aspects described herein;

FIG. 5D is a diagram illustrating a difference between an average numberof times days past due for the 20% most economic sensitive and the 20%least economic sensitive consumers within a risk score band, inaccordance with aspects described herein;

FIG. 6A is a diagram illustrating a difference between an average numberof months since the oldest trade line opened for the 20% most balancechange sensitive and the 20% least balance change sensitive consumerswithin a risk score band, in accordance with aspects described herein;

FIG. 6B is a diagram illustrating a difference between an average totalrevolving trade line balance for the 20% most balance change sensitiveand the 20% least balance change sensitive consumers within a risk scoreband, in accordance with aspects described herein;

FIG. 6C is a diagram illustrating a difference between an average numberof months since the most recent trade line for the 20% most balancechange sensitive and the 20% least balance change sensitive consumerswithin a risk score band, in accordance with aspects described herein;

FIG. 6D is a diagram illustrating a difference between an average amountpaid down on installment loans for the 20% most balance change sensitiveand the 20% least balance change sensitive consumers within a risk scoreband, in accordance with aspects described herein;

FIG. 6E is a diagram illustrating a difference between an average numberof times 90 days past due for the 20% most balance change sensitive andthe 20% least economic sensitive consumers within a risk score band, inaccordance with aspects described herein;

FIG. 7 is a diagram illustrating schematically the interplay ofpredictions, disruptions, and future entity behavior, in accordance withaspects described herein; and

FIG. 8 is a flowchart of a method for segmenting a population based onsensitivities and calculating a sensitivity of predictions about theentity's performance to potential future stressed conditions, inaccordance with aspects described herein.

When practical, similar reference numbers denote similar structures,features, or elements.

DETAILED DESCRIPTION

In certain aspects, the current subject matter relates to developmentand use of artificial intelligence and/or machine learning models tomake predictions about directionality and magnitude of the effects ofvarying future conditions on one or more performance metrics of a givenentity. Various currently available approaches in this field may relyupon large, randomizable data sets for an entire population. However,such approaches may be limited in their ability to make accuratepredictions based on specific current conditions for a given individualentity without requiring generalizing assumptions.

A beneficial approach, aspects of which are discussed herein, makes useof historical data (e.g., historical attribute values for multiple dataattributes relevant to the desired outcome prediction) in combinationwith machine learning-aided input data selection and recursive modelingbased on historical data from two or more time periods identified asrepresentative of differing levels of disruption or stress, as describedherein. For non-randomized data such as the historical attribute valuesused in implementations of the current subject matter, entities in thefirst and second sub-populations (e.g., those entities in the firstsub-population who experience a stressed or otherwise disruptedcondition subsequent to some initial date or dates at which historicalattribute values are available for these entities prior to the stressedor otherwise disrupted condition and those entities in the secondsub-population who experience a less stressed or otherwise disruptedcondition subsequent to some initial date or dates at which historicalattribute values are available for these entities prior to the stressedor otherwise disrupted condition) typically exhibit materially differentdistributions of their attribute values.

Consistent with implementations of the current subject matter,historical data are prepared for use in model development by identifyingfirst and second sub-populations which are “matched” in that an entityin the first sub-population has a sufficiently “similar” matched entityin the second sub-population. In this context, sufficient similarity ismeasured by the difference in a propensity score between two entities,one in the first sub-population and one in the second sub-population.The two sub-populations include entities whose historical data areavailable preceding and during time periods characterized by differingconditions. For example, the first sub-population is selected to includeentities for whom historical data are available preceding, during,and/or after a disruptive or otherwise stress-inducing event, condition,etc. The second sub-population is selected to include entities for whomhistorical data are available preceding, during, and/or after an event,condition, etc. that is either not disruptive or otherwisestress-inducing or is at least less disruptive or otherwisestress-inducing.

The predictive modeling approaches described herein can be applied to avariety of predictive scenarios. As illustrated in the process flowchart shown in FIG. 1A, a model generation process includes identifying,at 110, a population of entities (e.g., devices, machines, people,companies, etc.) for which historical (e.g., time series or otherwiseassociated with some form of temporal information) data values for aplurality of data attributes. The available historical data values for agiven entity in the population are either a) contemporaneous with (e.g.,prior to and during/after) either an event, time period, etc. thatcauses a binary disruption that is qualitatively expected to affect aperformance level, or the like of entities that experience it or b)contemporaneous with (e.g., prior to and during/after) an absence ofsuch an event, time period, etc. For purposes of this disclosure,reference to an “event” or “time period” is intended (unless otherwisespecifically narrowed) to refer to some duration of time (e.g.,instantaneous or occurring of some length of time) over which somedisruptive or less disruptive condition occurs for the relevant entityor entities for which historical data (e.g., historical attributevalues) are available.

At 120, the population of entities is divided into a firstsub-population and a second sup-population, where the firstsub-population experiences an historical event or condition (alsoreferred to as a first event in examples relating to a first event and asecond event), such as for example a disruption or a period ofdisruption, and the second sub-population does not experience the eventor condition. Alternatively, the second sub-population can experience asecond event that is considered less or even non-disruptive or thatotherwise serves as a “control” or baseline situation for purposes ofmaking a binary comparison between the effects of experiencingconditions consistent with the disruptive event and the effects of notexperiencing conditions consistent with the disruptive event orexperiencing conditions that are not as severe as the disruptive event.

The dividing of the population of entities into the first sub-populationand the second sub-population includes creating matched samples ofentities from the historical data such that two sub-populations havesimilar distributions of attribute values at the start of the binarycondition experiment. In other words, the first sub-population, whichexperiences a “disrupted” or “stressed” condition, and the secondsub-population of entities, which experiences a “normal” or “control”condition, are chosen such that the two sub-populations are similar intheir attribute distributions at or at least shortly before the time ofthe respective events (e.g., at the time of or at some time periodbefore the disruptive or stressed first event or the beginning of aperiod of low or no disruption or stress for the first sub-populationand at the time of or at some time before the non- or less-disruptive ornon- or less-stressed second event for the second sub-population).

Particularly when using multivariate historical data sets with largenumbers of data attributes that are not randomized and that are expectedto exhibit some degree of interrelationship between values of thevarious attributes, selection of entities to assign to the firstsub-population and the second sub-population generally requires morethan merely picking two groups of entities at different times withoutattempting to match historical attribute values for entities from thetwo groups to be paired. A technical challenge in this approach is thathistorical data is inherently non-randomized. In a randomized experimentin which it would be possible to freely generate similar attributedistributions for randomly selected subsets of a plurality of entitiessuch that each subset includes entities with comparable distributions ofhistorical attribute values of a plurality of data attributes. However,for non-randomized data such as the historical attribute values used inthe claimed invention, entities in the first and second sub-populationscan be expected in many instances to exhibit materially differentdistributions of their attribute values.

While traditional statistical techniques such as discriminant analysisor logistic regression analysis can in principle be used to performpropensity score-based matching subject to their statistical assumptions(including linearity in the attributes and no interactions between theattributes, as well as absence of multicollinearities and conformity toparametric assumptions), their application can be extremely tedious. Forexample, to capture nonlinearities and interactions between theattributes, many transformations of the raw attributes may need to be“manually” hypothesized by an analyst and then coded and tested, andmulticollinear attributes or transformations thereof may need to bedetected and removed prior to inclusion into the statistical model. Insuch a manual process, important transformations are readily overlooked.For non-randomized data such as historical attribute values, entities inthe first and second sub-populations (e.g., those entities in the firstsub-population who experience a disrupted or stressed condition, such asthe first event, and those entities in the second sub-population whoexperience a less disrupted or less stressed condition, such as thesecond event) may exhibit starkly different distributions of theirattribute values. Conventional approaches, including use of analysts andhuman-guided determination of the training data, are simply noteffectively applicable given the size and complexity of a large dataset, for example one characterized by hundreds of data attributes,millions of entities, nonlinear and non-parametric distributions, andinteractions between the attributes.

The inherent selection bias that can occur without properly preparingthe input data sets may be addressed by calculating a propensity scoreand using a propensity score matching technique. A propensity score maybe calculated for the entities in the original population usingsupervised machine learning. A supervised machine learning technology,such as for example that described in “Stochastic Gradient Boosting” (J.H. Friedman, Computational Statistics and Data Analysis, vol. 38, 2000,pp. 367-378) can be beneficial in developing propensity scores for thecreation of the first and second sub-populations with appropriatelysimilar data attribute value distributions. To calculate a propensityscore, the supervised machine learning can be trained on a curateddevelopment data set. Such a data set can include, for each entity, arecord of its attribute values and a target variable indicating thebinary condition experienced by the entity (e.g. disruption on/off).After training the model, each entity can be scored based on itsattribute values, resulting in a propensity score that models thelikelihood that a given entity experiences the disruption.

An approach using machine learning in this manner is not limited by theabovementioned assumptions (i.e., linearity, absence of interactions,parametric distributions, absence of multicollinearities). Instead, itcan address nonlinearities and interactions in a manner that humananalysis cannot match and can also effectively handle nonparametricdistributions and multicollinearities. The machine learning approachthus supports highly effective development of powerful propensityscores, such that propensity-score based matching may reduce or evenfully remove the abovementioned selection bias.

Based on the calculated propensity scores, entities in the originalpopulation are matched using a propensity score-based matchingtechnique. The matching provides a first sub-population of entities anda second sub-population of entities having similarly distributed valuesof the relevant data attributes and thereby allows reduction ofselection bias effects in the modeling applied to these data sets. Insome implementations of the current subject matter, a “pairwise matchingalgorithm” can be employed to identify pairs of entities that have verysimilar propensity score values (i.e. very similar likelihood ofexperiencing the disruption), but where one of the two entities wentthrough the disruption and the other did not.

With the data set prepared as noted above such that the first and secondsub-populations are sorted with sufficiently similar distributions ofdata attribute values to provide a “matched sample” of groups ofentities, a set of predictors are defined. These predictors include thematched entities' historical data attributes and a binary (0/1 for“normal”/“stressed”/disrupted) indicator variable. At 130, supervisedmachine learning techniques are used to model (e.g., using one or moreregression techniques) the entities' observed performances based onthese predictors. In other words, the current subject matter employssupervised machine learning analysis of observed performances (e.g., oneor more outcomes) of the entities of the two sub-populations duringand/or after the respective first and second events to produce a modelfor predicting outcomes based on an entity's data attributes and theevent (condition) indicator.

At 140, expected performances of the entities (in both the first andsecond sub-populations) under normal (e.g., less disrupted) and underdisrupted conditions are calculated using the models developed at 130.For each entity of the plurality of matched samples of entities, anexpected performance under both of the disrupted condition and the lessdisrupted condition is predicted using the modeling of the observedperformances. In some examples, this can be accomplished by varying thevalue of the binary indicator variable (e.g., predictors defined above)for an entity from 0 to 1, while keeping the entity's attribute valuesfixed.

At 150 a sensitivity value (e.g., Low, Medium, High) is calculated foreach matched entity by quantifying a difference between the expectedperformance under the disrupted condition and the expected performanceunder the non- or less disrupted condition. In some examples, thesensitivity value can be a score derived from a difference between afirst outcome score for the entity predicted at the disrupted conditiona second outcome score for the entity predicted at the less-disruptedcondition.

Using supervised learning techniques based on these calculatedsensitivity values and the historical attribute values for each entity,at 160 a sensitivity index model is generated. The resultant sensitivityindex model can be configured to produce a sensitivity index score forany entity based on input attribute values of one or more of theplurality of data attributes used in model development. The use ofsupervised machine learning enables accounting for the nonlinearitiesand non-randomized distributions of the input data. This model allows,among other potential benefits, robust predictions of an entity'ssensitivity to disruptions based on their current attribute values asinputs.

Additional benefits that may be realized from some implementations ofthe current subject matter include the improved “explainability” ofsensitivity predictions generated by the model. In certain applications,the ability to explain the sensitivity index values to users of a model,to affected entities, etc. can be quite important. Explainability mayalso facilitate engineering a model to be palatable to users, legallycompliant, and more robust (e.g. by enforcing one or more regulationsrelating to the model predictions, embedding contextual knowledge, ormitigating potential data biases).

In further aspects of the current subject matter, ensemble modelingtechniques can be applied. In this context, multiple sensitivity indexmodels are generated, where each model is based on differentpermutations of potential outcomes, disruptive events, etc. FIG. 1Bshows a flow chart 170 illustrating certain features consistent withthis aspect of the subject matter. At 175, a sensitivity model isgenerated per the discussion above in relation to FIG. 1A. This firstmodel is based on a first disruption type and/or a first target outcome.At 180, a plurality of additional models are generated per thediscussion above in relation to FIG. 1A. These additional models areeach based on one or more different disruption types and/or targetoutcomes, with the operations of FIG. 1A modified accordingly. Forexample, selection of the first and second sub-populations can betailored for each additional model to optimally capture a binary test ofa type of disruption or outcome. This can involve using differentevents, with different time periods and therefore different historicaldata attributes associated therewith, as well as changing one or morefeatures of the predicted outcome(s) for the entities being tested. At185, an ensemble sensitivity index is created, where the creatingincludes calculating a weighted combination of the outputs of thesensitivity models. The weighted combination can involve optimizingbased on analyzing the utility of each weighting for a decision maker,to provide the best possible predictions. Segmenting the outcomes in amanner consistent with this approach allows for a more focused andexplainable/understandable sensitivity score, at least because theunderlying models in the ensemble can be more tightly focused on typesof disruption and/or predicted outcomes, such that the binarydisrupted/non disrupted analysis can effectively incorporate differentlevels and types of disruptions and importance, relevance, etc. ofpredicted outcomes.

FIG. 2 depicts a block diagram illustrating a computing system 1400, inaccordance with some example embodiments. As shown in FIG. 2 , thecomputing system 200 can include a processor 210, a memory 220, astorage device 230, and input/output devices 240. The processor 210, thememory 220, the storage device 230, and the input/output devices 240 canbe interconnected via a system bus 1450. The processor 210 is capable ofprocessing instructions for execution within the computing system 200.In some implementations of the current subject matter, the processor 210can be a single-threaded processor. Alternately, the processor 210 canbe a multi-threaded processor. The processor 210 is capable ofprocessing instructions stored in the memory 220 and/or on the storagedevice 230 to display graphical information for a user interfaceprovided via the input/output device 240.

The memory 220 is a computer readable medium such as volatile ornon-volatile random-access memory (RAM) that stores information withinthe computing system 200. The storage device 230 is capable of providingpersistent storage for the computing system 200. The storage device 230can be a floppy disk device, a hard disk device, an optical disk device,or a tape device, or other suitable persistent storage means. Theinput/output device 240 provides input/output operations for thecomputing system 200. In some implementations of the current subjectmatter, the input/output device 240 includes a keyboard and/or pointingdevice. In various implementations, the input/output device 240 includesa display unit for displaying graphical user interfaces.

According to some implementations of the current subject matter, theinput/output device 240 can provide input/output operations for anetwork device. For example, the input/output device 240 can includeEthernet ports or other networking ports to communicate with one or morewired and/or wireless networks (e.g., a local area network (LAN), a widearea network (WAN), the Internet).

In some implementations of the current subject matter, the computingsystem 200 can be used to execute various interactive computer softwareapplications that can be used for organization, analysis and/or storageof data in various formats. The computing system 200 can be configuredto run one or more supervised machine learning algorithms consistentwith the features described herein. Alternatively, the computing system200 can be used to execute any type of software applications. Theseapplications can be used to perform various functionalities, e.g.,planning functionalities (e.g., generating, managing, editing ofspreadsheet documents, word processing documents, and/or any otherobjects, etc.), computing functionalities, communicationsfunctionalities, etc. The applications can include various add-infunctionalities or can be standalone computing products and/orfunctionalities. Upon activation within the applications, thefunctionalities can be used to generate the user interface provided viathe input/output device 240. The user interface can be generated andpresented to a user by the computing system 200 (e.g., on a computerscreen monitor, etc.).

The following examples are provided for context and to assist in betterunderstanding the current subject matter. They are not intended to belimiting unless features described are incorporated into the claimswhich follow this description.

In one example implementation of various features of the current subjectmatter relates to assessment of the sensitivity of predictions of drugefficacy to potential future disruptions a patient's health, such as forexample significant lifestyle changes, other health problems,disruptions in drug availability, etc. For illustrative purposes, thisexample relates to a cholesterol-control drug, but it will be understoodthat a variety of medical outcomes can be similarly modeled.

As an initial aspect of a method for training a sensitivity predictionmodel, historical health attribute data for a large population ofsubjects is processed. For the purposes of this example, the subjects(e.g., entities) can be pre-selected as a group having high cholesterolvalues. The population is further sorted into two sub-populations havingexperienced two different arms of a constructed historical experiment.For example, the first sub-population may have used cholesterolmedication for some period of time prior to ceasing use of themedication (e.g., due to unwanted side-effects, changes to differenttypes of treatment, loss of insurance coverage or other cost increase tothe drug, etc.). A second sub-population can be selected to includeindividuals who did not experience any such disruption in treatment.Data attributes of interest can include a wide variety of variables,including age, weight, body mass index, ethnic background,socio-economic factors, other health conditions, other medications beingtaken, and many others.

The groupings of subjects to be included in each sub-population can beidentified using a supervised machine learning calculation of apropensity score to determine if a sufficient number of subjects sharingsame or similar historical attribute values are available for the firstsub-population of entities and the second sub-population of subjects.This determining can involve calculating a propensity score and using apropensity score matching technique to identify matched samples ofsubjects who fit in the two respective sub-populations while overallcontributing to a similar distribution of data attribute values betweenthe two sub-populations.

Observed performances of the subjects in the first sub-populationconcurrent with and/or following the event based on the first historicalattribute values are modeled, as are observed performances of thesubjects in the second sub-population concurrent with and/or followingthe second event (or in general if the second event is merely an absenceof the first event) based on the second historical attribute values.This modeling can make use of one or more supervised machine learningtechniques.

Using the modeling of the two sub-populations based on their historicaloutcomes associated with their respective events, an expected outcome(e.g., performance, etc.) for each (i.e., from both sub-populations)subject is predicted under both conditions (e.g., experiencing the firstevent or the second event/not the first event). For each subject, asensitivity value or score is calculated as a quantified differencebetween the predicted outcome, performance, etc. for that subject undereach of the two modeled conditions (experiencing the first event or thesecond event/not the first event).

Then, the sensitivity values or scores for all of the subjects are used,along with their respective historical attribute values, for asupervised machine learning analysis that generates a sensitivity indexmodel useful in quantifying how sensitive a predicted health outcome fora given subject, for whom his or her own historical attribute values areavailable, is to a potential future disruption such as the first event.

Another example implementation of various features of the currentsubject matter relates to analysis and segmentation of entities based ontheir sensitivities to certain conditions. Using the sensitivitysegments, a risk scoring system can better detect high default riskentities and more accurately predict entity future behavior. In someexamples, approaches consistent with this embodiment may enablecalculation of economic or financial sensitivity index values forentities.

Risk scoring is widely used by banks and other financial institutionsfor assessing, and reporting, a measure of the creditworthiness ofindividuals. Often, risk scores are generated for an individual inassociation with assessing risk for a particular transaction, such asobtaining a mortgage or other loan, or opening up a new credit line suchas applying for a credit card. To generate a risk score, a riskmanagement reporting agency, typically at the request of a bank orfinancial institution, applies a modeling algorithm to the credit dataassociated with an individual.

Conventional techniques do not take into account how certain financialand economic disruptions may affect a consumer's future paymentperformance and their future risk score. That is, given a consumer'shistory, conventional techniques do not take into account whether a riskscore may move in a positive direction or negative direction.Accordingly, what is needed is a solution that provides a way toquantifying sensitivity of a prediction associated with a current riskscore to future disruptions, stressed conditions, etc. that could affecta consumer's future payment performance. Further, it can be beneficialto segment a seemingly homogenous population (e.g., a group of entitieswith similar conventional risk scores) into different groups to moreaccurately reflect their sensitivity to a future disruptive or otherwisestressed condition.

In some aspects, future substantial changes, or disruptions, toborrowers' situations following a date on which a convention risk scoreis calculated can have a substantial impact on payment performance thatis not predicted by risk scores. As one consequence, such disruptionscan lead to substantial discrepancies between predicted and actualfuture default odds. As another consequence, such changes can alsoreduce the rank ordering performance of the scores.

For example, for a given economic disruption, analysis of the resultingeconomic impact may indicate that actual default odds for a group ofconsumers in a homogeneous risk score band were substantially higher fora sub-group exposed after a scoring date to a recessionary economy, thanfor another sub-group exposed after the scoring date to a stableeconomy.

In another example, for a given disruption in financial obligations,analysis of the resulting economic impact may indicate that actualdefault odds for a group of consumers in a homogeneous risk score bandwere substantially higher for a sub-group who after a scoring dateincreased their credit card balances by substantial amounts (therebyincreasing their financial obligations), than for another sub-group whoafter the scoring date did not increase their card balances by asubstantial amount.

In some aspects, it may be desirable for lenders to identify those whoare not in a financially robust situation if they face an unexpected,unavoidable cost for an expensive medical procedure, or anotherunexpected expense. There are many sources and types of disruptions thatmight have an impact on an entity's loan repayment behavior, including,but not limited to: interest rate shocks, changes to income oremployment status, changes to social relationships, property loss,accidents, injuries and illnesses, etc. In general, it can be difficult,costly, and often quite impractical, to try to predict futuredisruptions with a high degree of confidence. Accordingly, it may bebeneficial for a scoring system to account for future disruptions thatare undetermined and unpredicted at a scoring date.

Disruption examples discussed herein relate to unfavorable changes tosituations, also referred to as “stress factors.” The disruptions,stress factors, etc. can apply equally to both positive or favorabledisruptions (e.g. job promotion, inheritance, lottery win) as tonegative or unfavorable disruptions. Typically, an entity's performanceis expected to worsen if an unfavorable disruption occurs, and theopposite might be expected when a favorable disruption occurs. However,it is possible that if an unfavorable disruption occurs, some entities'payment performance may not worsen, and some may actually improve. Forexample, certain individuals might redouble their efforts to repay theirdebt when conditions worsen, while other individuals may benefit fromdifficult conditions due to various factors. Similarly, if a favorabledisruption occurs, some entities' performance may not improve, and somemay actually worsen. For example, a windfall may seduce certainindividuals to live above their means and eventually experiencehardships as a consequence.

Through improved modeling and analysis it is possible to gain insightinto the variety of possible responses of entities to disruptions,without making assumptions regarding either directionality or magnitudeof the effect of disruptions on individual entities' paymentperformances. Accordingly, the entity segmentation for analysis ofeconomic sensitivity discussed herein may beneficially add flexibilityand improved accuracy to current risk scoring models not previouslyavailable. The benefit occurs in at least segmenting heterogeneousentities into “sensitivity segments” based on a sensitivity to adisruption/condition to more accurately predict future paymentperformance. The entities in any given sensitivity segment can besimilarly impacted by a certain type, or definition of, adisruption/condition.

Substantially worsening economic conditions, as exemplified by the GreatRecession, and amassing debt, as exemplified by rapidly growing creditcard balances, can be referred to as economic and financial stressfactors. A consumer may or may not be exposed to a certain stressfactor. Exposure to a stress factor may drive certain consumers torenege on their future credit obligations, whereas other consumersexposed to the same stress factor may hardly be affected. It may bebeneficial to measure this effect to more accurately predict futurepayment performance and reflect that prediction in a risk score. In someimplementations, a processor can implement a scoring system and createan ordinal scale of consumer sensitivities for each type, or definition,of a disruption or a stress factor. In some aspects, consumers can beranked and segmented according to their sensitivities.

Any number of sensitivity segments can be generated as desired withlesser or finer granularities and possibly non-equal segmentproportions. Segmentations with finer granularities can also beconstructed by incorporating other variables into the segmentdefinitions. For example a given sub-population grouped within arelatively small range of scores on a convention risk score model couldbe further sub-segmented into any number of sensitivity groups obtainedfrom the distribution of sensitivities within the particular score rangeof interest.

Having constructed stress-sensitivity segments for various types ofdisruptions, entities (e.g., consumers) can be more deeply and moreeasily understood and managed in terms of the risks they pose tolenders, by not only taking into account their risk scores, but inaddition, also calling out the extra risks due to impacts of possiblefuture disruptions. These extra risks increase for consumers who aremore sensitive to disruptions.

Knowledge of consumer sensitivities can enable lenders to takemitigating actions in order to reduce total risk, which arises in partis due to unpredicted disruptions. As an example, a lender worried aboutthe next recession might reduce exposure to consumers with high economicsensitivities and increase exposure to consumers with low economicsensitivities. The lender might consider combinations of risk scorevalues and sensitivity segments to create preference rankings whereby aconsumer with a marginally lower conventional risk score yet a favorablylow sensitivity might be preferred over a consumer with slightly higherrisk score yet an unfavorably high sensitivity. Preferences might beexpressed through marketing targeting, through accepting or rejecting acredit line request, through settings of loan limits, through pricing,etc.

In some aspects, it is possible to define an entity's sensitivity to adisruption or stress factor in the framework of the Rubin causal model,as the difference between potential performances for the entity whensubjected to alternative situations or conditions, namely a “normal”condition and a “disrupted” or “stressed” condition. As such, anentity's sensitivity is an individual-level causal effect of a binarycondition on future payment performance. In this framework, normal andstressed conditions appear as two arms of a thought experiment. Inreality an entity can only travel along one arm of the experiment forwhich the entity's performance is then observed. Performance for theuntraveled arm cannot be observed.

FIG. 3 is a diagram 300 of an individual's sensitivity under twodifferent conditions (e.g., a normal and stressed condition). As shown,in FIG. 3 , the individual, X_(Joe), 902, can have certain attributevalues at the outset of an experiment, also referred to as the “scoringdate.” The experiment attempts to predict Joe's performance under twodifferent conditions, a normal condition and a stressed (e.g., economicrecession or downturn) condition. At the end of the experiment, theindividual's (Joe's) potential performance under normal conditions isrepresented as Y1 904 and Joe's potential performance under stressedconditions is represented as Y2 906. Joe's sensitivity to the stressedcondition (e.g., disruption or stress factor) can be defined based onthe difference between Y1 904 and Y2 906.

Expanding from the example of FIG. 3 , in some aspects, if certainstatistical and econometric conditions hold on a sample of developmentdata consisting of entities' attributes at a scoring date and of theexperimental conditions subject to which entities' performances wereobserved, then it is possible to estimate individual-specific causaleffects on ordinal scales. In some implementations, estimatingsensitivities to financial stress factors or other disruptions asindividual-specific causal effects, can leverage natural experiments ina transparent and fail-safe manner.

For example, a method of estimating individual sensitivities can includea first step of determining if there are a sufficient number of entitiesthat share the same or similar attribute values at scoring date yetsubsequently travel through different arms of the experiment. Forexample, if a large number of entities share one or more attributevalues or similar attribute values (e.g., income, payment history,outstanding balances, number of inquiries, etc.), and those entitiesalso experience different disruptions or stress factors (e.g., halfundergo normal conditions and half undergo stressed condition). In someaspects, determining which entities share the same or similar attributevalues can be based on a propensity score. In some implementations thepropensity score can be calculated using any propensity score matchingtechnique. For example, a propensity score can be calculated using atechnique described in the publication “The Central Role of thePropensity Score in Observational Studies for Causal Effects” Biometrika70 (1): 41-55, (1983) by Paul Rosenbaum and Donald Rubin.

If the answer is ‘no’ then sensitivity estimation cannot be accomplishedwith confidence (fail-safe). If the answer is ‘yes’, then a sensitivityestimating system may, in a second step, create a matched sample ofentities where a first sub-population of entities travels along thenormal condition arm and a second sub-population of other entitiestravels along the stressed condition arm, such that the twosub-populations are similar in their attribute distributions at thescoring date.

Next, in a third step, the sensitivity estimating system can definepredictors comprised of the matched entities' attributes at the scoringdate and a binary (0/1 for “normal”/“stressed”) indicator variable. Thesensitivity estimating system can use supervised machine learningtechniques to regress the entities' observed performances based on thesepredictors. In a fourth step, for each matched entity, the sensitivityestimating system can predict expected entities' performances undernormal and under stressed conditions, by varying the value of the binaryindicator variable (e.g., predictors defined in the third step) from 0to 1, while keeping the entity's attributes fixed. Compute sensitivityvalue (e.g., Low, Medium, High) of each matched entity by differencingnormal and stressed predictions.

In a fifth step, the sensitivity estimating system can use supervisedmachine learning techniques to regress the entities' sensitivity valuesbased on the entities' observable attributes at the scoring date. Forexample, the regression may indicate that entities in at a certainincome group have a higher sensitivity than entities in a differentincome group. In a sixth step, the sensitivity estimating system can usethe regression model from the fifth step to predict the sensitivities ofany entities of interest. The entities of interest referred to the sixthstep can be new entities, such as new customers, or they can be existingentities whose attribute values may change over time, thus allowingsensitivities of entities, which need not to remain constant over time,to be regularly updated based on the latest data available on theentities. For example, a new customer can have certain attribute valuesthat match with, or are similar to, other entities used in thesensitivity estimating system that had a Low economic sensitivity index(ESI). Accordingly, the new customer may also be assigned a Low ESI.

In some implementations, a proof-of-concept model for economicsensitivity described herein can be based on US credit bureau datacollected during two starkly contrasting phases of the recent USeconomic cycle. Payment performance for a stable economy (“normalcondition”) can be collected during the 2-year window starting withscoring date October 2013 and ending October 2015. Payment performancefor a recessionary economy (“stressed condition”) can be collectedduring the 2-year window starting with scoring date October 2007 andending October 2009 which falls into the time of the Great Recession.The binary (“normal”/“stressed”) indicator was accordingly defined as:‘0’ for a first group of consumers whose attributes were collected inOctober 2013 and who subsequently performed under normal conditions; and‘1’ for a second group of consumers whose attributes were collected inOctober 2007 and who subsequently performed under stressed conditions.

In some aspects, a proof-of-concept model for credit card balance changesensitivity described herein can be based on US credit bureau datacollected and combined from multiple scoring dates across a recenteconomic cycle, including both stable and recessionary performanceperiods. In this way, the balance change sensitivity model is not tiedto a specific economic condition but captures averaged behaviors fromacross various economic conditions. Payment performance for“non-increasers” (“normal condition”) was collected for consumers whodidn't increase their card balances by more than $100, or decreasedtheir card balances, over a “balance change window” of 6 monthsfollowing a scoring date. Payment performance for “increasers”(“stressed condition”) was collected for consumers who increased theircard balances by more than $2,000 over the balance change window. In allcases, payment performance was collected over a 2-year window followingthe balance change window.

FIG. 4 is a time diagram 1000 that illustrates this longitudinal studydesign. The binary (“normal”/“stressed”) indicator was accordinglydefined as: ‘0’ for a first group of consumers who didn't increase theircard balances by more than $100, or decreased their card balances, overthe balance change window, with their performances observed under these“normal” conditions; and ‘1’ for a second group of consumers whoincreased their card balances by more than $2,000 over the balancechange window, with their performances observed under these “stressed”conditions. As shown in FIG. 4 , month 0 is the scoring date whichbegins the experiment. The two groups are represented as two lines, thefirst group is the top line 4010 and the second group is represented bythe bottom line 4020. At month 6, the study can measure the creditbalance change for all participants and define the two groups (e.g.,define the two lines 4010 and 4020). During months 6-30 (“performanceperiod”), the study can measure the performance of the two groups overtime. At month 30, the study can perform an analysis of the two groupsover the performance period and generate payment performance statisticsbased on the analysis.

During both model developments (e.g., economic sensitivity and balancechange sensitivity) the study found sufficient numbers of entities thatshared similar attribute values at the scoring date (month 0) andsubsequently traveled through different arms of their experiments, (i.e.performed under “normal” and under “stressed” conditions). The studythen used supervised machine learning techniques to regress the entitiesand calculated the economic sensitivities and the balance changesensitivities based on the entities' observable attributes at thescoring date for a large and representative sample of US consumers whoregularly access consumer credit.

FIG. 5A is a diagram illustrating a difference between an average numberof inquiries for the 20% most economic sensitive and the 20% leasteconomic sensitive consumers within the risk score band of 678 to 682.FIG. 5B is a diagram illustrating a difference between an average totaltrade line balance for the 20% most economic sensitive and the 20% leasteconomic sensitive consumers within the risk score band of 678 to 682.FIG. 5C is a diagram illustrating a difference between an average numberof months since the most recent trade line for the 20% most economicsensitive and the 20% least economic sensitive consumers within the riskscore band of 678 to 682. FIG. 5D is a diagram illustrating a differencebetween an average number of times 90 days past due for the 20% mosteconomic sensitive and the 20% least economic sensitive consumers withina risk score band, in accordance with aspects described herein.

As shown in FIGS. 5A-D, having more credit inquiries, having highertrade line balances, having more recently a new trade line opened, andhaving lower average number of times days past due, are all associatedwith having higher economic sensitivity.

Empirically, data analysis can find that the default rate more thandoubles during the stressed economic period versus the normal economicperiod for the 20% most sensitives in a given score band, whereas thedefault rate may hardly vary across economic conditions for the 20%least sensitives in this score band. Such information can be useful tocompanies deciding between consumers with similar risk scores butdifferent economic sensitivity scores.

Similarly, the sub-population within the score band from 678 to 682 maybe further sub-segmented, or alternatively sub-segmented, into balancechange sensitivity quintiles based on the distribution of economicsensitivities within this FICO® Score band. In the illustrative example,the risk score band (e.g. from 678 to 682) is relatively narrow, suchthat from the traditional risk scoring perspective, this sub-populationof entities would be regarded as a homogeneous risk pool. However, asillustrated below, the lowest and the highest balance change sensitivityquintile segments differ substantially in their attribute distributions.

FIG. 6A is a diagram illustrating a difference between an average numberof months since the oldest trade line opened for the 20% most balancechange sensitive and the 20% least balance change sensitive consumerswithin the risk score band of 678 to 682. FIG. 12B is a diagramillustrating a difference between an average total revolving trade linebalance for the 20% most balance change sensitive and the 20% leastbalance change sensitive consumers within the risk score band of 678 to682. FIG. 6C is a diagram illustrating a difference between an averagenumber of months since the most recent trade line for the 20% mostbalance change sensitive and the 20% least balance change sensitiveconsumers within the risk score band of 678 to 682. FIG. 12D is adiagram illustrating a difference between an average amount paid down oninstallment loans for the 20% most balance change sensitive and the 20%least balance change sensitive consumers within the risk score band of678 to 682. FIG. 6E is a diagram illustrating a difference between anaverage number of times 90 days past due for the 20% most balance changesensitive and the 20% least balance change sensitive consumers withinthe risk score band of 678 to 682.

As shown in FIGS. 6A-E, having less maturation time of oldest creditline, having higher revolving balances, having more recently a new tradeline opened, having made lower down payments on installment loans, andhaving lower average number of times 90 days past due, are allassociated with having higher balance change sensitivity.

Empirically, data analysis can find that the default rate variesconsiderably more across balance stress conditions for the 20% mostbalance change sensitive consumers than for the 20% least balance changesensitive consumers in a given score band. Such information can beuseful to companies deciding between consumers with similar risk scoresbut different balance change sensitivity scores.

While economic and balance change sensitivities are described herein, itis possible to calculate other consumer sensitivities. For example,sensitivity scores can reflect the interplay between predictions of anykinds of behaviors of entities (not necessarily their future paymentperformance, and predictions not necessarily based on credit bureaudata), disruptions of any kind (as long as data on the disruptions arecollected), and entities' actual future behaviors. In some aspects,consumers could be segmented into groups that differ in terms of impactof health insurance loss on future investment decisions, or groups thatdiffer in terms of impact of adopting a cholesterol-lowering medicationon future levels thereof (as discussed above in the earlier exampleimplementation), or groups that differ in terms of impact of enrollmentin a driver education program on future driving skills, etc.

FIG. 7 is a diagram 700 illustrating schematically the interplay ofpredictions, disruptions, and future entity behavior. As illustrated inFIG. 7 , a predictive model may base its prediction 710 of an entity'sfuture behavior on a variety of data sources and data attributes 704associated with the entity at a certain time. The model may alsoconsider sensitivities to a variety of disruptions 702 to determine aneffect of a given disruption to the entity that would otherwise beunaccounted for by the predictive model.

FIG. 8 is a flowchart of a method 800 for segmenting a population basedon sensitivities and a calculating risk score based on the segmentedsensitivities. In various implementations, the method 800 (or at least aportion thereof) may be performed by the computing system 200, otherrelated apparatuses, and/or some portion thereof. In some aspects, thecomputing system 200 may be regarded as a server and/or a computer.

Method 800 can start at operational block 810 where the computing system200, for example, can receive one or more attributes associated with afirst entity. Method 800 can proceed to operational block 820 where thecomputing system 200, for example, can calculate a sensitivity index forthe first entity based on the one or more attributes. In someimplementations, calculating a sensitivity index can additionally oralternatively involve the computing system 200, for example, creating amatched sample of entities, the entities sharing at least one attributevalue of the one or more attributes, the matched sample of entitiescomprising a first sub-population of the entities experiencing a firstcondition and a second sub-population of the entities experiencing asecond condition, the first sub-population different from the secondsub-population. In some implementations, calculating a sensitivity indexcan additionally or alternatively involve the computing system 200, forexample, calculating, for each entity of the matched sample of entities,a sensitivity value associated with the entity, the calculatingcomprising subtracting an expected performance under the first conditionwith an expected performance under the second condition. In someimplementations, calculating a sensitivity index can additionally oralternatively involve the computing system 200, for example, segmenting,by the computer processor, any sample of entities into two or moresegments based on the sensitivity value of each entity, the sensitivityindex comprising one of the two or more segments.

Method 800 can proceed to operational block 830 where the computingsystem 200, for example, can calculate a second risk score for the firstentity based on the sensitivity index and the first risk score of theentity. Method 800 can proceed to operational block 830 where thecomputing system 200, for example, can output the second risk score to auser interface. While the operational blocks of method 800 areillustrated and described in a particular order, each of the operationblocks can be performed in any order.

Certain aspects of the subject matter of the current application arerelated to features described in co-pending U.S. publication no.2019/0130481A1, the disclosure of which is incorporated herein byreference in its entirety.

One or more aspects or features of the subject matter described hereincan be realized in digital electronic circuitry, integrated circuitry,specially designed application specific integrated circuits (ASICs),field programmable gate arrays (FPGAs) computer hardware, firmware,software, and/or combinations thereof. These various aspects or featurescan include implementation in one or more computer programs that areexecutable and/or interpretable on a programmable system including atleast one programmable processor, which can be special or generalpurpose, coupled to receive data and instructions from, and to transmitdata and instructions to, a storage system, at least one input device,and at least one output device. The programmable system or computingsystem may include clients and servers. A client and server aregenerally remote from each other and typically interact through acommunication network. The relationship of client and server arises byvirtue of computer programs running on the respective computers andhaving a client-server relationship to each other.

These computer programs, which can also be referred to as programs,software, software applications, applications, components, or code,include machine instructions for a programmable processor, and can beimplemented in a high-level procedural and/or object-orientedprogramming language, and/or in assembly/machine language. As usedherein, the term “machine-readable medium” refers to any computerprogram product, apparatus and/or device, such as for example magneticdiscs, optical disks, memory, and Programmable Logic Devices (PLDs),used to provide machine instructions and/or data to a programmableprocessor, including a machine-readable medium that receives machineinstructions as a machine-readable signal. The term “machine-readablesignal” refers to any signal used to provide machine instructions and/ordata to a programmable processor. The machine-readable medium can storesuch machine instructions non-transitorily, such as for example as woulda non-transient solid-state memory or a magnetic hard drive or anyequivalent storage medium. The machine-readable medium can alternativelyor additionally store such machine instructions in a transient manner,such as for example as would a processor cache or other random accessmemory associated with one or more physical processor cores.

To provide for interaction with a user, one or more aspects or featuresof the subject matter described herein can be implemented on a computerhaving a display device, such as for example a cathode ray tube (CRT), aliquid crystal display (LCD) or a light emitting diode (LED) monitor fordisplaying information to the user and a keyboard and a pointing device,such as for example a mouse or a trackball, by which the user mayprovide input to the computer. Other kinds of devices can be used toprovide for interaction with a user as well. For example, feedbackprovided to the user can be any form of sensory feedback, such as forexample visual feedback, auditory feedback, or tactile feedback; andinput from the user may be received in any form, including, but notlimited to, acoustic, speech, or tactile input. Other possible inputdevices include, but are not limited to, touch screens or othertouch-sensitive devices such as single or multi-point resistive orcapacitive trackpads, voice recognition hardware and software, opticalscanners, optical pointers, digital image capture devices and associatedinterpretation software, and the like.

The subject matter described herein can be embodied in systems,apparatus, methods, and/or articles depending on the desiredconfiguration. The implementations set forth in the foregoingdescription do not represent all implementations consistent with thesubject matter described herein. Instead, they are merely some examplesconsistent with aspects related to the described subject matter.Although a few variations have been described in detail above, othermodifications or additions are possible. In particular, further featuresand/or variations can be provided in addition to those set forth herein.For example, the implementations described above can be directed tovarious combinations and subcombinations of the disclosed featuresand/or combinations and subcombinations of several further featuresdisclosed above. In addition, the logic flows depicted in theaccompanying figures and/or described herein do not necessarily requirethe particular order shown, or sequential order, to achieve desirableresults. Other implementations may be within the scope of the followingclaims.

What is claimed is:
 1. An improved computer-implemented machine learningsystem comprising: one or more processors coupled to one or more datastorage devices to process or store data associated with knownattributes for a plurality of entities subjected to a plurality of knownevents; and a machine learning model implemented as a data structurestored in the one or more data storage devices, the machine learningmodel trained to observe performances of the plurality of entities in afirst sub-population and a second sub-population, the machine learningmodel associated with a set of predictors and at least a binaryindicator variable corresponding to a first entity subjected to a firstevent experience by members of the first sub-population, the machinelearning model trained to predict an expected performance for the firstentity based on at least one of (1) a known attribute associated withthe first entity in relation to the first event and (2) a value of thebinary indicator variable associated with the first event, members ofthe first sub-population and the second sup-population beingdistinguishable based on whether one or more events have beenexperienced by the members of the first sub-population and the secondsup-population, the members of the first sub-population experiencing thefirst event and the members of the second sub-population experiencingthe second event, wherein an expected performance of the first entitysubjected to the first event is determined based on a sensitivity valuedetermined for the first entity by quantifying an expected performanceof a member of the first sub-population subjected to the first eventaccording to a sensitivity index generated based on differentpermutations of potential outcomes for the plurality of known events,and a treatment being applied to the first entity, responsive to thedetermined expected performance.
 2. The system of claim 1, wherein thefirst-subpopulation and the second sub-populations optimally capture abinary test of a type of outcome for the sensitivity index based ondifferent events with different time periods associated with differentknown attributes.
 3. The system of claim 2, wherein the different knownattributes are contemporaneous to at least one of (1) the differentevents causing a binary disruption in the first entity's status quo, (2)an absence of the first event, and (3) the second event being lessdisruptive than the first event.
 4. The system of claim 3, wherein thedifferent known attributes have values that are non-randomized, suchthat the first entity experiencing the first event and a second entityexperiencing the second event exhibit materially different distributionsof said values.
 5. The system of claim 1, wherein the sensitivity indexis generated based on at least a first disruption type or a first targetoutcome.
 6. The system of claim 1, wherein a known attribute isdetermined based on historical data associated with a population ofentities having been subjected to an event.
 7. The system of claim 4,wherein the population is divided into the first sub-population and thesecond sub-population based on a propensity score calculated for pairsof matched entities in the population.
 8. The system of claim 1, whereinthe first event comprises a first medical treatment and the second eventcomprises a second medical treatment not including the first medicaltreatment.
 9. The system of claim 1, wherein the historical datacomprises one or more of age, weight, body mass index, ethnicbackground, socio-economic factors, and health condition.
 10. The systemof claim 6, wherein the sensitivity index predicts how sensitive apredicted health outcome for the first entity is depending on apotential future disruption in the first medical treatment.